Download spark_python_task workspace files behind --download-spark-python-files flag#5892
Queued
janniklasrose wants to merge 7 commits into
Queued
Conversation
…generate job This re-applies commit b2e7e6f (PR #5799), which was reverted in #5837 (commit 353cb5a). `bundle generate job` again downloads workspace files referenced by `spark_python_task` and rewrites them to a relative path, like it already does for notebooks. Git-sourced files and cloud URIs are left untouched. The behaviour is gated behind a feature flag in a follow-up commit.
…flag Downloading workspace files referenced by a spark_python_task is now opt-in via a new --download-spark-python-files flag on bundle generate job (default off). A Python file often imports sibling files that the downloader does not capture, so downloading only the entry point can produce a job that fails at runtime with missing imports; this was the reason #5799 was originally reverted in #5837. The behaviour is threaded through the Downloader as a functional option (WithSparkPythonFiles), so the pipeline, app, and import callers are unaffected.
janniklasrose
temporarily deployed
to
test-trigger-is
July 10, 2026 09:56 — with 
GitHub Actions
Inactive
GitHub Actions
Inactive
janniklasrose
temporarily deployed
to
test-trigger-is
July 10, 2026 09:56 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Collaborator
Integration test reportCommit: 06965a8
8 interesting tests: 4 RECOVERED, 4 SKIP
Top 10 slowest tests (at least 2 minutes):
|
…generate-jobs-download-script-with-flag
janniklasrose
temporarily deployed
to
test-trigger-is
July 11, 2026 20:30 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
janniklasrose
temporarily deployed
to
test-trigger-is
July 11, 2026 20:30 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
andrewnester
approved these changes
Jul 15, 2026
janniklasrose
added this pull request to the merge queue
Jul 17, 2026
Any commits made after this event will not be merged.
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Jul 17, 2026
…generate-jobs-download-script-with-flag
janniklasrose
enabled auto-merge
July 17, 2026 12:32
janniklasrose
added this pull request to the merge queue
Jul 17, 2026
Any commits made after this event will not be merged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Re-applies #5799 (reverted in #5837) and puts the behaviour behind a new opt-in flag.
bundle generate jobonly downloaded notebook tasks; files referenced byspark_python_taskwere left as absolute/Workspace/...paths, so the source file was never downloaded and the config wasn't portable. This PR restores the download+rewrite path (reusing the samemarkFileForDownloadhelper as pipeline libraries) but gates it behind a new--download-spark-python-filesflag onbundle generate job, defaulting to off. Git-sourced files (source: GIT) and cloud URIs (dbfs:/,s3:/,adls:/,gcs:/) are left untouched.The flag is threaded through the
Downloaderas a functional option (WithSparkPythonFiles), so the pipeline, app, and import callers are unaffected.Why
#5799 was reverted in #5837 because a Python file often imports sibling files that the downloader does not capture, so downloading only the entry point can produce a job that fails at runtime with missing imports — whereas leaving the absolute workspace path alone "just works". Making the download opt-in keeps the default safe while letting users who know their
spark_python_taskis self-contained pull the file into their bundle.Commits
NEXT_CHANGELOG.mdconflict resolved).--download-spark-python-files— the new flag, functional-option plumbing, tests, and regenerated help/changelog.Tests
bundle/generate/downloader_test.go: the download+rewrite path (with the option), the skipped cases (cloud URI,source: GIT), and a new test asserting the default-off behaviour makes no requests.acceptance/bundle/generate/spark_python_task_jobexercising the full CLI with the flag: a workspace-file task is downloaded and rewritten, adbfs:/cloud-URI task is preserved. Identical output on bothterraformanddirectengines.bundle generate job --helpgolden file.This pull request and its description were written by Isaac, an AI coding agent.